Library¶
In [ ]:
import pandas as pd
import numpy as np
from bertopic import BERTopic
import sys
sys.path.append('../../../Util')
import ShowGraphs as sg
2 Baseline Summary¶
Clustering Approach¶
- Parameter Setting:
- Embedding Model: all-MiniLM-L6-v2
- Representation Model: keyBERTInspired, MaximalMarginalRelevance
- Count Vectorizer
- CtfIDF
- UMAP: 20 neighbors, 10 components
- HDBSCAN: 150 min cluster size
- Zero-Shot Classification on cluster name
- Outliers-reduction with Cosine Similarity: 0.46 threshold
Clustering Results¶
Clusters Retrieved: 68 in which the most important ones concern:
- Drug sales (marijuana, cocaine, xanax, pills, meth, fentanyl)
- Bitcoin
- Scammers and seller reviews
- Marketplace advertising
- Purchase reviews
- Drug purchases
- Orders
- Closed sites (empire market,
- Scams
- Sold passwords
- Hacker attacks
- Opsec questions
- Document and credit card forgery
- Chat links
Performance Metrics:
- Silhouette Score: 0.50
- Davies-Bouldin Score: 0.87
- Coherence-Score: 0.57
- Dos Score: 0.06
- %Outliers: 0.30 (19k/65k)
Code¶
In order to avoid importing the whole BERTopicUtils.py module, and thus the linked libraries, I have only brought back the function needed to make predictions, it is the same as in the module
In [2]:
def predict_topic(topic_model: BERTopic, sentence: list, num_classes: int = 5, custom_labels: bool = False) -> pd.DataFrame:
"""
Predict the topic of a sentence using the BERTopic model.
:param topic_model: The BERTopic model.
:param sentence: The sentence to predict the topic of.
:param num_classes: The number of classes to return.
:param custom_labels: Whether to use custom labels.
:return: A DataFrame with the predicted topics.
"""
# Transform the sentence
_, pr = topic_model.transform(sentence)
# Get the top indices
top_indices = np.argsort(pr[0])[::-1][:num_classes]
# Get the top topics
if custom_labels:
top_topics = [(topic_model.get_topic(i), pr[0][i], topic_model.custom_labels_[i+1]) for i in top_indices]
else:
top_topics = [(topic_model.get_topic(i), pr[0][i], topic_model.generate_topic_labels()[i+1]) for i in top_indices]
# Create a DataFrame with the results
df_finals = pd.DataFrame(top_topics, columns=['Topic', 'Probability', 'Label'])
# Extract the words and sentence
df_finals['Words'] = df_finals['Topic'].apply(lambda topic: [word for word, _ in topic])
df_finals['Sentence'] = sentence * len(df_finals)
return df_finals
Visualize CSV Files¶
In [3]:
topic_model = BERTopic.load("../../../Analyze_files/CombiningAnalysisCompleteDataset/ThreadAnalysis/Models/topic_model_all-MiniLM-L6-v2_150_20n")
In [4]:
descr_topic = pd.read_csv('CSV68Topic/description_topic.csv')
document_topic = pd.read_csv('CSV68Topic/document_topic_proba.csv')
topics_over_time = pd.read_csv('CSV68Topic/topic_over_time_3.csv')
Topic Description¶
In [4]:
print(descr_topic.shape[0])
descr_topic.head()
68
Out[4]:
| Topic | Count | BERTopic_Name | Representation | Representative_Docs | Custom_Name_GenAI | Custom_Name_Zero_Shot | |
|---|---|---|---|---|---|---|---|
| 0 | 56 | 242 | 56_apollon_apollon market_market_apollon apollon | ['apollon' 'apollon market' 'market' 'apollon ... | ['back apollon' 'apollon' 'apollon market'] | Apollon Market | market - apollon - apollon market |
| 1 | 24 | 544 | 24_coke_coke vendor_best coke_uk coke | ['coke' 'coke vendor' 'best coke' 'uk coke' 'u... | ['fire coke' 'coke vendor ' 'coke vendor'] | Coke Topic | cocaine |
| 2 | 62 | 290 | 62_alprazolam_powder_alprazolam powder_flualpr... | ['alprazolam' 'powder' 'alprazolam powder' 'fl... | ['usa domestic alprazolam powder mxe apvp inbo... | Alprazolam Powder | powder - alprazolam - alprazolam powder |
| 3 | 0 | 5021 | 0_weed_cannabis_cart_thc | ['weed' 'cannabis' 'cart' 'thc' 'review' 'hash... | ['product vendor review ml lemon kush wax vape... | Cannabis Weed Vape Cart Reviews | marijuana |
| 4 | 16 | 998 | 16_review_vendor review_review vendor_vendor | ['review' 'vendor review' 'review vendor' 'ven... | ['xpost danknation vendor review sunaero multi... | Vendor Reviews Product | reviews vendor |
In [6]:
sg.plot_topic_distribution(descr_topic, figsize=(12, 12))
In [5]:
sg.plot_topic_percentage_distribution(descr_topic, figsize=(12, 6))
In [12]:
sg.create_wordclouds(topic_model, num_topics=68, cols=3, width=800, height=500)
Document Topic Description¶
In [44]:
print(document_topic.shape[0])
document_topic.head()
46629
Out[44]:
| Document | Topic | Probability | Created_on | BERTopic_Name | |
|---|---|---|---|---|---|
| 0 | naturalmeds apollon | 56 | [1.19642027e-307 6.58965022e-308 7.28015586e-3... | 2020-01-09 | 56_apollon_apollon market_market_apollon apollon |
| 1 | redemption btc giveaway coke sub dread member ... | 24 | [0.0065961 0.00376317 0.00431397 0.00327901 0... | 2020-01-09 | 24_coke_coke vendor_best coke_uk coke |
| 2 | flubromazolam sample giveaway | 62 | [8.33571265e-308 7.43009529e-308 8.20561023e-3... | 2019-11-06 | 62_alprazolam_powder_alprazolam powder_flualpr... |
| 3 | cigarette tobacco replica | 0 | [0.04007918 0.00797617 0.01040953 0.00632349 0... | 2020-01-09 | 0_weed_cannabis_cart_thc |
| 4 | requiring image image review | 16 | [0.00675313 0.00730991 0.006758 0.00472002 0... | 2019-11-06 | 16_review_vendor review_review vendor_vendor |
In [5]:
result_df = sg.extract_max_probabilities(document_topic)
In [11]:
sg.plot_avg_prob_or_freq(result_df, 'Max_Probability', figsize=(20, 10))
In [13]:
sg.plot_boxplot(result_df, 'Max_Probability', figsize=(20, 10))
In [6]:
sg.plot_probability_distribution(result_df, 'Max_Probability')
In [ ]:
sg.create_wordclouds(document_topic, num_topics=68, cols=3, is_model=False, width=1000, height=600)
In [ ]:
sg.create_topic_network_graphs(document_topic, max_words=50, min_occurrences=10, cols=2, window_size=10)
